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(19) 



EuropaiSffnes Patentamt 
European Patent Office 
Office europeen des brevets 



CM 
< 



CM 
O) 
CO 

o 

Q_ 
LU 



BUFFER 
FULLNESS 



1000 



500 



100" 



STREAM 1 AU 1 




FIG. 2A 





AU 1 STREAM 2 
AU 



>AU 2/ 

iX^ 



AU 1 



"V 
STUFFING 
BJTS 




T, Uisi DTS NEXT 

DTS2 



TIME 



Printed by Jouve (FR), 18. rue Saint-Denis. 7S001 PARIS 



1 



EP0 692 911 A2 



Description 

BACKGROUND OF THE INVENTION 

This invention concerns the processing of video sig- 
nals encoded according to the standard adopted by the 
Moving Picture Experts Group (MPEG) and in particular, 
a method of splicing an MPEG encoded data stream 
from a first source to an MPEG encoded data stream 
from a second source in a manner that ensures that no 
video data is lost when the combined image is repro- 
duced. 

In the United States a standard has been proposed 
for digitally encoded high definition television signals. 
This standard is essentially the same as the MPEG-2 
standard, proposed by the Moving Picture Experts 
Group (MPEG) of the International Standards Organiza- 
tion (ISO). This standard is described in a draft interna- 
tional standard (DIS) publication entitled Information 
Technology - Generic Coding of Moving Pictures and As- 
sociated Audio, Recommendation H.262" which is avail- 
able from the ISO and which is hereby incorporated by 
reference for its teaching on the MPEG-2 systems stand- 
ard "1SO/IEC 13818-1" and for its teachings on the 
MPEG-2 digital video encoding standard "ISO/lEC 
13818-2." 

The MPEG-2 standard is actually several different 
standards. In MPEG-2 several different profiles are de- 
fined, each corresponding to a different level of complex- 
ity of the encoded image. For each profile, different levels 
are defined, each level corresponding to a different im- 
age resolution. One of the MPEG-2 "standards", known 
as Main Profile, Main Level is intended for encoding vid- 
eo signals conforming to existing television standards 
(i.e., NTSC and PAL). Another "standard", known as 
Main Profile, High Level is intended for encoding 
high-definition television images. Images encoded ac- 
cording to the Main Profile, High Level standard may 
have as many as 1 , 1 52 active lines per image frame and 
1 ,920 pixels per line. It is this standard which is currently 
being implemented as the United States HDTV standard. 

The MPEG-2 standard defines a complex syntax 
which contains a mixture of data and control information. 
Some of this control information is used to enable the 
signals having several different formats to be covered by 
the standard. These formats define images, having dif- 
fering numbers of picture elements (pixels) per line, dif- 
fering numbers of lines per frame or field and differing 
numbers of frames or fields per second. In addition, the 
basic syntax of the MPEG-2 Main Profile defines the 
compressed MPEG-2 bit stream representing a se- 
quence of images in six layers, the sequence layer, the 
group pictures layer/the picture layer, the slice layer, the 
macro block layer, and the block layer. Each of these lay- 
ers is introduced with control information and "stuffing" 
characters. The stuffing characters are inserted as need- 
ed to ensure that the data rate of the input data stream 
matches the rate at which pictures are displayed. Finally, 



other control information, also known as side informa- 
tion, (e.g. frame type, macroblock pattern, image motion 
vectors, coefficient zig-zag patterns and dequantization 
information) are interspersed throughout the coded bit 
5 stream. 

To effectively receive the digital images, a decoder 
must recognize the control portions, extract the neces- 
sary control information, and use the extracted data to 
process the video signal information. One piece of infor- 
10 mation that is specified for each sequence layer is the 
video buffering verifier (VBV) buffer size. The VBV buffer 
size value specifies a number of bits of input data from 
the video sequence which must be stored in the input 
buffer of the decoder before the video sequence may be 
is decoded. If this number of bits is stored when the decod- 
ing operation begins, the input buffer will neither become 
too full (overflow) or become empty (underflow) during 
the processing of the video sequence. 

The VBV buffer size value assumes that the input 
20 buffer is empty when the new sequence data is received 
or, at least that all of the data in the input buffer will be 
removed before the contents of the input buffer reach the 
VBV buffer size value. To ensure that the VBV buffer size 
value does not cause the input buffer to overflow or un- 
2$ derflow, it is preferred in the MPEG standard that any 
image splicing be done on a sequence boundary. 

This may be unacceptable, however, for video edi- 
tors who would like to switch from one scene to another 
after a particular picture has been displayed, irrespective 
30 of whether that picture is at a sequence boundary. It may 
also be undesirable for broadcasters who wish to insert 
short program segments, for example station identifica- 
tion messages or advertisements into an HDTV program 
at arbitrary intervals. 
35 If splicing is not done at a sequence boundary then 
the data already resident in the input buffer may not be 
read from the buffer for processing until the buffer over- 
flows due to the new data from the inserted sequence. 
Alternatively, the data already resident in the input buffer 
40 may cause the decoder to prematurely indicate that the 
VBV buffer size criterion has been met. In this instance, 
the decoder may cause the input buffer to either overflow 

or underflow in the processing of the stored data. 

J? One method of solving this problem is proposed in 
45 the MPEG-2 standard DIS ISO/lEC 13818-1. By this 
method, certain access units, or pictures, in the data 
stream are marked with a SEAMLESS_SPLICE_FLAG. 
This flag is used to mark safe locations for splicing an 
inserted sequence into the main sequence. As with the 
so conventional splicing method, however, there are only a 
limited number of access units which satisfy the criteria 
to have this flag set. Furthermore, this splicing method 
limits the bit rate of the inserted stream to a certain level 
to meet its criteria and the inserted stream can not be a 

55 variable bit rate stream. 

Another method would be to define a format within 
the MPEG standard which allows for easy insertion and 
provide a flag identifying this format. This method, how- 
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ever, cannot handle incompatible formats and reduces 
the robustness of the MPEG standard. 

SUMMARY OF THE INVENTION 

5 

The present invention is embodied in a method of 
processing a main MPEG encoded sequence to allow 
another sequence to be inserted at any picture boundary 
in the main sequence. According to the present inven- 
tion, the main and inserted MPEG sequences are ana- 10 
lyzed to determine a number of stuffing characters to in- 
sert in the data stream after the selected picture bound- 
ary and before the inserted sequence. The stuffing char- 
acters are ignored during the decoding operation and al- 
low the input buffer to be emptied to a level compatible is 
with the inserted sequence. This number stuffing char- 
acters is determined from the data rates of the main and 
inserted sequences and the intervals between respec- 
tive points in time corresponding to the start of the inser- 
tion and the next decoding time stamps in both the main 20 
and inserted sequences. The method inserts this 
number of stuffing characters after the sefected splice 
point and then inserts the inserted sequence. ^jj^ 

BRIEF DESCRIPTION OF THE DRAWINGS 25 

Figure 1 is a data diagram which illustrates an inser- 
tion operation according to the present invention. 

Figures 2a and 2b are graphs of buffer fullness ver- 
sus time which are useful for explaining the operation of 
the present invention for fixed data rate MPEG-2 encod- 
ed signals. 

Figures 3a and 3b are graphs of buffer fullness ver- 
sus time which are useful for explaining the operation of 
the present invention for variable data rate MPEG-2 en- 
coded signals. 

Figure 4 is a block diagram of circuitry suitable for 
use in an HDTV or MPEG-2 encoder and decoder which 
operates in accordance with the present invention. 

Figure 5 is a flow-chart diagram of a process which 
extracts parameter values from the main and inserted 
data stream. 

Figure 6 is a flow-chart diagram of a process which 
calculates a number of stuffing bits to be inserted from 
the extracted parameters. 

Figure 7 is a flow-chart diagram of an insertion proc- 
ess which inserts the number of stuffing bits calculated 
in Figure 6 between the insertion point between in main 
sequence and the start of the inserted sequence. 

Figures 7a, 7b and 7c are graphs of buffer fullness so 
versus time which are useful for explaining the operation 
of the present invention when the variable k, in Figure 7, 
has a value of one. 

DETAILED DESCRIPTION ss 

Figure 1 is a data diagram which illustrates the splic- 
ing operation. In this figures, a sequence 112 is inserted 



into a sequence 110 of the main HDTV signal to form a 
modified sequence 1 1 0'. As shown in Figure 1 , the main 
MPEG sequence 110 includes a sequence header 112, 
including a sequence start code, several access units 
(pictures) 116 and a sequence end code 114. For the 
sake of simplicity, the group of pictures layer, which is 
between the sequence layer and the picture layer is not 
shown in Figure 1. The sequence to be inserted, 112, 
also includes a sequence header 118, several access 
units 122 and a sequence end code 120. 

As defined in the MPEG standard, each sequence 
start code may be preceded by a string of zero-valued 
stuffing bits. In the MPEG standard, these bits represent 
null information which is ignored by the decoder and is 
not written into the input buffer. The stuffing bits are used 
to ensure that the rate at. which data is being supplied 
matches the rate at which images, derived from the data, 
are being displayed. While these stuffing bits are being 
received, the decoder is reading data from the input buff- 
er for display. Thus, the amount of data held in the input 
buffer is reduced while the stuffing bits are being re- 
ceived. 

The combined sequences are shown as the modi- 
fied data stream 110' of Figure 1. This data stream in- 
cludes the sequence header 112, access units 116, a 
modified sequence header 118' access units 122 and a 
modified sequence end code 120'. The modified se- 
quence start code 118' includes a sequence end code, 
which marks the end of the first part of the previous main 
sequence, a number, NSTUFF, of stuffing bits, the se- 
quence start code for the inserted sequence and the 
modified header for the inserted sequence. The bit-rate 
field of the original header for the inserted sequence is 
modified to be FFFF hexadecimal, indicating a variable 
bit-rate sequence. This change is made only if the bit rate 
of the inserted sequence is different from that of the main 
sequence and the original inserted sequence was a con- 
stant bit-rate sequence. 

The method according to the present invention splic- 
es the start of one sequence to an arbitrary access unit 
in another sequence. The insertion of the sequence is 
completed only after the remaining part of the original 
sequence is spliced to the inserted sequence in the same 
way that the inserted sequence was spliced to the main 
sequence. Thus, the modified sequence header 120' 
also includes a sequence end code, some stuffing bits 
and the modified sequence header for the main se- 
quence. As with the modified sequence header 118', the 
sequence header 120' may have its bit-rate field 
changed to FFFF hexadecimal, indicating a variable 
bit-rate sequence, if the bit-rates of the main and inserted 
sequences differ and if the original main sequence used 
a constant bit-rate. 

The number of stuffing bits to be added is deter- 
mined by the method of the subject invention as de- 
scribed below with reference to Figures 5 and 6. The ac- 
tual splicing operation is performed as described below 
with reference to Figures 2, 3 and 7. 
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Figures 2a and 2b are graphs of buffer fullness ver- 
sus time which illustrate the problem addressed by the 
present invention and the method by which the problem 
is solved. The solid line in Figure 2a shows part of the 
data stream for the main sequence, STREAM 1 , while the s 
solid line in Figure 2b shows part of the data stream for 
the inserted sequence, STREAM2. In the exemplary em- 
bodiments of the invention, these data streams are 
bit-streams. Vertical transitions in these Figures indicate 
times at which data is being read from the buffer. It is a 10 
convention in these types of diagrams that data is mod- 
eled as being read from the buffer instantaneously. Por- 
tions of the curve which have slanted lines indicate times 
at which data is being written into the buffer. The slope 
of these lines is proportional to the data rate of the data ts 
stream. 

The sequence STREAM2 is to be inserted into the 
sequence STREAM1 at time T,, immediately after ac- 
cess unit AUfflf the maximum input buffer size used by 
this example is 1000 then, if STREAM2 were inserted 20 
into STREAM* at time T,, then the buffer would overflow 
before the VBV buffer size (i.e. 500) for STREAM2 had 
been reached?To prevent this from occurring, the meth- 
od according to the present invention inserts a number, 
NSTUFF, of stuffing bits between the AU1 at time T, and 25 
the start of STREAM2. This is indicated by the dashed 
line in Figure 2a. Thus, after T,, instead of following the 
solid line, the buffer fullness for the MPEG bit stream fol- 
lows the dashed line. The values T last , DTS1 , T next , DT- 
S next and DTS2 are used, as described below to calcu- 30 
late the optimal value for NSTUFF. While the stuffing bits 
are being inserted into the data stream, no new data is 
added to the buffer. New data, representing the inserted 
sequence is added when the slope of the dashed line is 
n on -zero. 

It is noted, from Figures 2a and 2b that the rate at 
which data is provided by STREAM2 is less than the rate 
at which it is ^provided by STREAM1. Nonetheless, the 
first portion ofhhe data of STREAM2, provided between 
the decoding^ime stamps DTS1 to DTS next , is provided 
at the same rate as the sequence STREAM1 while the 
portion of STREAM2 which is provided after DTS next is 
provided at the same rate as the original STREAM2, 
shown in Figure 2b. 

The graphs in Figures 2a and 2b illustrate the oper- 
ation of an exemplary embodiment of the inventive meth- 
od when the sequences are encoded to have a fixed data 
rate. Figures 3a and 3b illustrate the operation of the 
method when the sequences are encoded to have a var- 
iable data rate. Items in these Figures which correspond 
to items in Figures 2a and 2b are indicated by a prime 
mark ('). 

Figure 4 is a block diagram showing portions of an 
MPEG-2 or HDTV splicer and an MPEG-2 or HDTV de- 
coder according to the present invention. The splicer in- 
cludes a processor 412, which is coupled to receive the 
main and insert MPEG-2 HDTV signals, STREAM1 and 
STREAM2, respectively. Each of these signals may be 



generated by a conventional HDTV encoder, such as is 
described in U.S. Patent No. 5,294,974 entitled, HIGH 
DEFINITION VIDEO ENCODING SYSTEM HAVING 
COLOR-SENSITIVE QUANTIZATION, which is hereby 
incorporated by reference for its teaching on HDTV en- 
coders. Alternatively, these signals may be generated by 
an MPEG-2 encoder such as that described in ISO/IEC 
13818-2 DIS, which is hereby incorporated by reference 
for its teachings on MPEG-2 encoders. '* 

The processor 412 is coupled to two buffers, BStest 
414 and BSdec 416. These buffers are preferably iden- 
tical. The buffer 414 is used to gather information from 
which the value NSTUFF is calculated. The actual splic- 
ing is performed using the buffer 41 6. The spliced signal 
is then transmitted from buffer 41 6 to the input buffer 424 
of the decoder 420. The decoder 420 also includes a 
processor 422 which retrieves data from the input buffer 
424 in order to reproduce a moving image. 

Figure 5 is a flow-chart diagram which illustrates the 
process that controls the processor 41 2 in gathering in- 
formation about the sequences STREAM1 and 
STREAM2. This process begins at step 51 0. At step 51 2, 
the processor initializes the buffer 41 4 by setting its write 
address pointer to the beginning of the buffer. At step 
514, the processor sends STREAM1 to the buffer at the 
data rate specified in the sequence header for 
STREAM1 . It is noted that in the exemplary embodiment 
of the invention, both STREAM1 and STREAM2 are 
stored signals (e.g. recovered from digital video tape re- 
corders). Thus, the signals may be accessed twice, once 
to determine splicing parameters and again to perform 
the actual splicing operation. For splicing two signals 
which are not stored but come, for example, from a sat- 
ellite feed, a compensating delay (not shown) may be 
needed so that the splicing parameters may be deter- 
mined before the actual splicing operation occurs. 

At step 514, the process determines if the insertion 
point, immediately after the picture represented by ac- 
cess unit AU1 has been stored into the buffer 414. Until 
AU1 is received, every access unit that is received, is 
removed from the buffer 41 4 at the time indicated by the 
decoding time stamp (DTS) field of the access unit. In 
other words, the data is removed from the buffer 414 in 
the same sequence and with the same timing that it 
would be removed from an input buffer by a decoder, 
such as the decoder 420 shown in Figure 4. Even when 
each access unit is removed, the sequence STREAM1 
continues to be provided at its proper data rate, as indi- 
cated by step 520, and control is transferred to step 516 
to determine if AU1 has been received. 

Once access unit AU1 has been received, control 
transfers to step 522 to determine values for T, ast and 
N last . As shown in Figure 2a, T, ast is the difference in time 
between when AU? was received and DTS1 , the decod- 
ing time stamp for AU1 , the time at which AU1 is to be 
read from the input buffer for processing. 

Next, at step 524, the process determines values for 
DTS next , T next , and N next . DTS next is the time to the next 
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decoder time stamp after DTS1. In the MPEG standard, 
the decoder time stamps occur at regular intervals. Ac- 
cordingly, DTS next may be determined by calculating the 
difference between DTS1 and the preceding decoder 
time stamp and then adding this difference to DTS1 . Al- 
ternatively, this value may be determined by monitoring 
STREAM*!, extracting the DTS values from the access 
unit headers which follow AU1 until a DTS other than 
DTS1 is encountered. As shown in Figure 2a, T next is 
calculated by subtracting DTS1 from DTS next . Likewise, 
N next , the number of bits of data which will be stored into 
the buffer during the time interval T next , is calculated by 
multiplying T next by the data rate, R1, of STREAM1 or 
by any rate which is less than or equal to the maximum 
data stream rate, Rmax[p.1], as defined in ISO/I EC 
13818-1 and ISO/IEC 13818-2. This data rate is availa- 
ble from the sequence header. If STREAM1 is a variable 
rate data stream, as shown in Figure 3a, R1 is any data 
rate which is less than or equal to the maximum data rate 
for the data stream, R max [p.1], as defined in ISO/IEC 
13818-1 and ISO/IEC 13818-2. 

At step 526, once these values have been deter- 
mined, the process again empties the buffer 414 and, at 
step 528, begins storing data from STREAM2 into buffer 
414. This data is stored into the buffer at the bit rate of 
STRE AM2. As shown by the loop including step 532, the 
data is stored until a time corresponding to the decoder 
time stamp for the first access unit of STREAM2. This 
time is designated DTS2*. The amount of data stored in 
buffer 41 4 at DTS2* is designated as N new and this value 
is saved by the process at step 534. At step 536, the 
process determines whether STREAM2 is a fixed rate 
stream or a variable rate stream. If it is a variable rate 
stream, control is transferred to step 538 to determine a 
value R2, representing the bit rate of STREAM2. In this 
instance, R2 is assigned the average value of the data 
rates over the time interval defined by DTS2*. At step 
540, the process of collecting the parameters ends. 

After the parameters have been determined, the 
processor 41 2 calculates the value NSTUFF. This proc- 
ess starts at step 610 of Figure 6. At step 61 2, a tempo- 
rary variable K is set to zero. Next, step 614 calculates 
a value N sp | C , representing an amount of data which is 
to be written into the buffer having the data rate R2. 

In some splices, the amount of data represented by 
N new may not be able to be stored into the buffer, at the 
data rate R1 , during the time interval T next . This situation 
is handled by the loop which includes the steps 614, 616 
and 618. At step 61 4, a trial value for N splc is determined 
as the product of K, T next and R2. At step 616, this trial 
value is compared to the quantity N new minus N| ast minus 

N next- 

If, however, N spIc is greater than or equal to N new 
minus N, ast minus N next at step 616, then control is trans- 
ferred to step 620 to determine the value of NSTUFF ac- 
cording to equation (1 ). 



NSTUFF = N tast + N next + N splc - N new (1 ) 

At step 622, the process ends. The inventors have 
determined that if this number of stuffing bits are added 
5 before the inserted sequence STREAM2, there will be 
no overflow of the input buffer when the data stream is 
received. 

After a suitable value of NSTUFF has been deter- 
mined, the processor 412 executes the process shown 

10 in Figure 7 to splice STREAM2 to STRE AM1 . This proc- 
ess begins at step 710 by emptying the buffer BSdec, 
416, shown in Figure 4. At step 712, the processor 412 
sends data from STREAM 1 into buffer 416 at the data 
rate of STREAM1. At step 714, the process determines 

75 if the last bit of AU1 has been written into buffer 416. if 
not, the process continues to provide data from 
STREAM1 until AU1 has been stored as indicated by 
steps 716 and 718, removing any AU's prior to AU1 at 
intervals corresponding to their decoder time stamps 

20 (DTS's). 

After AU1 has been stored, the process, at step 720 
removes any AU's in the buffer 416 which precede and 
include AU1 at the corresponding decoding time stamp 
of each AU. Starting from the time that AU1 has been 

25 stored, the process, at step 722 sends NSTUFF stuffing 
bits to buffer 41 6 at the bit rate R1 . At step 724, the proc- 
ess determines if all NSTUFF of the stuffing bits were 
sent before the time DTS next . If not, control transfers to 
step 734 which determines if all NSTUFF of the stuffing 

30 bits were sent at the time DTS next . If this condition is not 
met, step 736 is executed to send the remaining stuffing 
bits at rate R2 to buffer 416. 

If, however, at step 724, it was determined that the 
NSTUFF stuffing bits had been sent before DTS next , step 

35 726 is executed which begins sending the initial part of 
STREAM2 to buffer 416 at bit-rate R1. At step 728, the 
process determines if the variable K has a value of zero. 
If not, control transfers to step 738 in which data from 
STREAM2 is sent to buffer 416 at the bit-rate R2. Step 

40 738 is also executed after step 734 if it is determined that 
NSTUFF bits have been sent at time DTS next and after 
step 736. After step 738, step 740 is executed which re- 
moves AU2 from buffer 416 at a time corresponding to 
DTS2. 

45 Figures 7a, 7b and 7c are graphs of buffer fullness 
versus time which show a splice in which K has a value 
of one. In Figure 7c, it is noted that between the time 
indicated by DTS nexl and the time indicated by 'DTS2, 
the bit rate is R2, the bit rate of STREAM2, if STREAM2 

50 is a constant bit-rate sequence, and it is the average 
bit-rate during T2* (shown in Figure 7b) if STREAM2 is 
a variable bit-rate sequence. 

If, however, at step 728, it is determined that the var- 
iable K is equal to zero, then at step 730, first access unit 

55 of STREAM2, AU2, is removed from buffer. After step 
730 or step 740, step 732 is executed which continues 
to provide data from STREAM2, at the bit-rate of 
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STREAM2. This is the last step of the splicing process. 

As shown in Figure 4, data is removed from buffer 
416 and applied to the input buffer 424 of the decoder 
420. The buffer 424 ignores the stuffing bits, effectively 
delaying the application of the STREAM2 data until suf- 
ficient data has been removed from the input buffer 424, 
by the processor 422, to make room for the STREAM2 
data. 

While the invention has been described in terms of 
an exemplary embodiment, it is contemplated that it may 
be practiced as described above in accordance with the 
following claims. 
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A method of splicing a first and second data streams 
representing respective first and second encoded 
high-definition television signals at a selected pic- 
ture boundary in the first data stream using a buffer, 
wherein each of the first and second data streams 
has a data rate and a plurality of decoding time 
stamps representing times at which data is retrieved 
from the buffer, the method comprising the steps of: 

analyzing the first data stream to determine an 
amount of null information to be inserted in the first 
data stream after the selected picture boundary, 
wherein The amount of null data is determined from 
the data rates of the first and second data streams 
and the decoding time stamps in the first and second 
data streams; 

transmitting the first data stream, up to the 
selected picture boundary to the buffer; 

transmitting the determined amount of null 
information to the buffer; 

transmitting the second data stream to the 
buffer; and 

retrieving the spliced data stream from the 
buffer. 

A method according to claim 1 , wherein the step of 
transmitting the second data stream to the buffer 
includes the steps of: 

transmitting a first portion of the second data 
stream to the buffer at the data rate of the first data 
stream; and 

transmitting a second portion of the second 
data stream to the buffer at the data rate of the sec- 
ond data stream. 

A method according to claim 2, wherein the data 
streams are bit-streams encoded according to a 
standard developed by the motion-picture experts 
group (MPEG) and the amount of null information is 
a number of stuffing bits to be inserted before a 
sequence start code of the second data stream. 
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of null information is determined by the equation: 
NSTUFF = N |ast + N next + N sp|c - N new 

where Nj ast represents a number of bits in the 
first data stream between the time that the selected 
picture is stored into the buffer and the decoder time 
stamp at which the selected picture is retrieved from 
the memory; N next represents a number of bits of the 
first data stream that would be provided in an interval 
between two decoder time stamps; N sptc represents 
a number of bits of the first bit-stream that would be 
provided in an integer number of intervals between 
decoder time stamps of the first data stream, the 
number of bits being greater than or equal to N new , 
the number of bits in the second data stream occur- 
ring before the first decoder time stamp of the sec- 
ond data stream. 
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4. A method according to claim 3, wherein the amount 
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